Search for: All records

We apply idealized scatterplot distributions to the sliding threshold of observation for numeric evaluation (STONE) curve, a new model assessment metric, to examine the relationship between the STONE curve and the underlying point‐spread distribution. The STONE curve is based on the relative operating characteristic (ROC) curve but is developed to work with a continuous‐valued set of observations, sweeping both the observed and modeled event identification threshold simultaneously. This is particularly useful for model predictions of time series data as is the case for much of terrestrial weather and space weather. The identical sweep of both the model and observational thresholds results in changes to both the modeled and observed event states as the quadrant boundaries shift. The changes in a data‐model pair's event status result in nonmonotonic features to appear in the STONE curve when compared to an ROC curve for the same observational and model data sets. Such features reveal characteristics in the underlying distributions of the data and model values. Many idealized data sets were created with known distributions, connecting certain scatterplot features to distinct STONE curve signatures. A comprehensive suite of feature‐signature combinations is presented, including their relationship to several other metrics. It is shown that nonmonotonic features appear if a local spread is more than 0.2 of the full domain or if a local bias is more than half of the local spread. The example of real‐time plasma sheet electron modeling is used to show the usefulness of this technique, especially in combination with other metrics.

 

We have developed a new procedure for combining lists of substorm onset times from multiple sources. We apply this procedure to observational data and to magnetohydrodynamic (MHD) model output from 1–31 January 2005. We show that this procedure is capable of rejecting false positive identifications and filling data gaps that appear in individual lists. The resulting combined onset lists produce a waiting time distribution that is comparable to previously published results, and superposed epoch analyses of the solar wind driving conditions and magnetospheric response during the resulting onset times are also comparable to previous results. Comparison of the substorm onset list from the MHD model to that obtained from observational data reveals that the MHD model reproduces many of the characteristic features of the observed substorms, in terms of solar wind driving, magnetospheric response, and waiting time distribution. Heidke skill scores show that the MHD model has statistically significant skill in predicting substorm onset times.

 

There is considerable evidence that current sheet scattering (CSS) plays an important role in isotropic boundary (IB) formation during quiet time. However, IB formation can also result from scattering by electromagnetic ion cyclotron waves, which are much more prevalent during storm time. The effectiveness of CSS can be estimated by the parameter, the ratio of the field line radius of curvature to the particle gyroradius. Using magnetohydrodynamic and empirical models, we estimated the parameterKassociated with storm time IB observations on the nightside. We used magnetic field observations from spacecraft in the magnetotail to estimate and correct for errors in theKvalues computed by the models. We find that the magnetohydrodynamic and empirical models produce fairly similar results without correction and that correction increases this similarity. Accounting for uncertainty in both the latitude of the IB and the threshold value ofKrequired for CSS, we found that 29–54% of the IB observations satisfied the criteria for CSS. We found no correlation between the correctedKand magnetic local time, which further supports the hypothesis that CSS played a significant role in forming the observed IBs.

 

Recent studies have found that even during quiet times, observed proton isotropic boundaries (IBs) are often projected to the region of high adiabaticity parameter (K≈30), whereis the ratio of magnetic field line radius of curvature to the particle gyroradius. This contradicts the accepted hypothesis that current sheet scattering (CSS) is the dominant mechanism of IB formation becauseK≈8 would be expected for this mechanism. We used magnetohydrodynamic simulations and empirical models to computeKfor 30‐keV proton IB observations within 3 hr of local midnight. We found that neither class of model reliably estimatesKunless supported by magnetic field observations in the current sheet. magnetohydrodynamic simulations produced higherKvalues than expected for CSS (K = 15–30), and empirical models gave lower values (K < 4). We obtained reliable estimates ofKby controlling for the accuracy of the normal component and the gradient of the radial component in the neutral sheet, using observations from three Time History of Events and Macroscale Interactions during Substorms satellites. For the first time, we demonstrated that both these variables should be taken into account for the accurate estimation of the curvature radius. This greatly reduced the spread ofKvalues, indicating that much of the previous spread was due to errors in the magnetic field but also that these errors can be controlled. Most of the corrected values fall within the expected range for CSS, supporting the hypothesis that the IB's were formed by CSS. Accounting for all model results, we obtain an average corrected value ofK = 6.0.

 

A new model validation and performance assessment tool is introduced, the sliding threshold of observation for numeric evaluation (STONE) curve. It is based on the relative operating characteristic (ROC) curve technique, but instead of sorting all observations in a categorical classification, the STONE tool uses the continuous nature of the observations. Rather than defining events in the observations and then sliding the threshold only in the classifier/model data set, the threshold is changed simultaneously for both the observational and model values, with the same threshold value for both data and model. This is only possible if the observations are continuous and the model output is in the same units and scale as the observations, that is, the model is trying to exactly reproduce the data. The STONE curve has several similarities with the ROC curve—plotting probability of detection against probability of false detection, ranging from the (1,1) corner for low thresholds to the (0,0) corner for high thresholds, and values above the zero‐intercept unity‐slope line indicating better than random predictive ability. The main difference is that the STONE curve can be nonmonotonic, doubling back in both thexandydirections. These ripples reveal asymmetries in the data‐model value pairs. This new technique is applied to modeling output of a common geomagnetic activity index as well as energetic electron fluxes in the Earth's inner magnetosphere. It is not limited to space physics applications but can be used for any scientific or engineering field where numerical models are used to reproduce observations.

 

The space physics community continues to grow and become both more interdisciplinary and more intertwined with commercial and government operations. This has created a need for a framework to easily identify what projects can be used for specific applications and how close the tool is to routine autonomous or on-demand implementation and operation. We propose the Application Usability Level (AUL) framework and publicizing AULs to help the community quantify the progress of successful applications, metrics, and validation efforts. This framework will also aid the scientific community by supplying the type of information needed to build off of previously published work and publicizing the applications and requirements needed by the user communities. In this paper, we define the AUL framework, outline the milestones required for progression to higher AULs, and provide example projects utilizing the AUL framework. This work has been completed as part of the activities of the Assessment of Understanding and Quantifying Progress working group which is part of the International Forum for Space Weather Capabilities Assessment. 

The Community Coordinated Modeling Center has been leading community‐wide space science and space weather model validation projects for many years. These efforts have been broadened and extended via the newly launched International Forum for Space Weather Modeling Capabilities Assessment (https://ccmc.gsfc.nasa.gov/assessment/). Its objective is to track space weather models' progress and performance over time, a capability that is critically needed in space weather operations and different user communities in general. The Space Radiation and Plasma Effects Working Team of the aforementioned International Forum works on one of the many focused evaluation topics and deals with five different subtopics (https://ccmc.gsfc.nasa.gov/assessment/topics/radiation‐all.php) and varieties of particle populations: Surface Charging from tens of eV to 50‐keV electrons and internal charging due to energetic electrons from hundreds keV to several MeVs. Single‐event effects from solar energetic particles and galactic cosmic rays (several MeV to TeV), total dose due to accumulation of doses from electrons (>100 keV) and protons (>1 MeV) in a broad energy range, and radiation effects from solar energetic particles and galactic cosmic rays at aviation altitudes. A unique aspect of the Space Radiation and Plasma Effects focus area is that it bridges the space environments, engineering, and user communities. The intent of the paper is to provide an overview of the current status and to suggest a guide for how to best validate space environment models for operational/engineering use, which includes selection of essential space environment and effect quantities and appropriate metrics.